feat(cua-driver): expose element AX actions in structured elements by hqhq1025 · Pull Request #2622 · trycua/cua

hqhq1025 · 2026-07-28T07:23:39Z

Summary

get_window_state already collects each element's exposed AX actions. AXNode.actions is populated during the walk (ax/tree.rs) and rendered into tree_markdown:

INDENT- [N] AXRole "Title" [value="..." actions=[...]]

The structured elements array never carried them. A caller reading structuredContent therefore cannot tell which action a given element supports, and has to guess a name for perform_action — where a wrong guess is indistinguishable from an element that legitimately refused.

This is the same gap that was already closed for `value`

The comment above the value field in build_elements_array_with_token states the problem and the fix:

the value is shadowed and invisible to a caller reading the structured side — it only showed up in tree_markdown, forcing a markdown grep to verify what landed. Emit it explicitly so the verify-then-escalate loop can read the typed text structurally.

actions has exactly that shape today: present in the markdown, absent from the structured side, forcing a markdown grep for something the walk already knows.

It also matches what the tool description already promises about where new information goes:

new fields will only be added to the structured side.

Change

  if let Some(parent) = node.parent_element_index {
      entry["parent_index"] = serde_json::json!(parent);
  }
+ if !node.actions.is_empty() {
+     entry["actions"] = serde_json::json!(node.actions);
+ }

Before / after for one row:

// before
{"element_index": 4, "element_token": "s0a1b:4", "role": "AXButton",
 "label": "Send", "depth": 3, "frame": {...}}

// after
{"element_index": 4, "element_token": "s0a1b:4", "role": "AXButton",
 "label": "Send", "depth": 3, "frame": {...},
 "actions": ["AXPress", "AXShowMenu"]}

Additive only — no existing field changes shape. The key is omitted when an element exposes no actions, so rows for inert nodes (AXStaticText and friends) do not grow.

Platform scope

macOS only. The Linux (AT-SPI) and Windows (UIA) element builders do not collect an equivalent action list today, so there is nothing to emit there yet — the field is optional per platform rather than a new cross-platform contract. If it's preferred to land this as a cross-platform field, I'm happy to look at what AT-SPI / UIA expose and extend it in a follow-up.

Test

cargo test -p platform-macos — 184 passed, 0 failed
cargo fmt -p platform-macos -- --check — clean
cua-contract-gen all --check — generated manifest already up to date

New test asserts both directions: an actionable element carries its action list, and an element with no actions omits the key rather than emitting an empty array.

Note

Independent of #2608 and #2621. Also split out of #2210, which is now a tracking draft.

`get_window_state` already collects each element's exposed AX actions — `AXNode.actions` is populated by the walk and rendered into `tree_markdown` as `actions=[...]`. The structured `elements` array never carried them, so a caller reading `structuredContent` has no way to know which action a given element supports and must guess a name for `perform_action`. A wrong guess is indistinguishable from an element that legitimately refused. This is the same gap that was already closed for `value`. The comment above that field says it plainly: the value is shadowed and invisible to a caller reading the structured side — it only showed up in `tree_markdown`, forcing a markdown grep to verify what landed. Emit it explicitly so the verify-then-escalate loop can read the typed text structurally. `actions` has exactly that shape: present in the markdown, absent from the structured side, forcing a markdown grep for something the walk already knows. The key is omitted when the element exposes no actions, so rows for inert nodes (AXStaticText and friends) do not grow. Additive only — no existing field changes shape. macOS only for now: the Linux (AT-SPI) and Windows (UIA) element builders do not collect an equivalent action list today, so there is nothing to emit there yet. The field is therefore optional per platform rather than a new cross-platform contract. Verified: - `cargo test -p platform-macos`: 184 passed, 0 failed - `cargo fmt -p platform-macos -- --check`: clean - `cua-contract-gen all --check`: generated manifest already up to date

chatgpt-codex-connector · 2026-07-28T07:23:45Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

hqhq1025 · 2026-07-28T08:48:12Z

Context for this one is in #2608 (comment) — it is part of a three-way split of #2210, and each piece is independent of the other two. Reviewable on its own; no dependency on the others landing.

hqhq1025 · 2026-07-28T15:16:14Z

@f-trycua — this is the last of the three-way split of #2210; #2608 and #2621 are both merged now.

It is the smallest of the three: one file, +40 −1, and no new concept. The AX walk already collects each element's exposed actions and already renders them into tree_markdown; the structured elements array just never carried them, so a caller reading the preferred structured output has to guess an action name for perform_action or fall back to parsing markdown. This closes the same gap that was already closed for value, with the same reasoning as the comment above that field.

platform-macos: 184 passed, 0 failed.

Also, on #2210 itself: I am not going to keep rebasing it. After the split, and after dropping the three things I said I would not carry forward (the UUID token format, allow_pixel_fallback(via_token), and node_identity), what is left is essentially the fix for your review's [P1] point 2 — the destructive token consumers that reacquire the macOS element by raw index instead of the generation/identity-aware retained node. That stands on its own, so I will send it as a fourth single-file PR against current main rather than dragging 19 files through another merge. I will close #2210 once that is up.

No rush on this one either.

…oss platforms The Linux, macOS, and Windows accessibility walkers already collect action names, but none of the platform get_window_state tools surfaced them in the structured elements array. This caused callers to see missing/empty actions even though the underlying element was actionable. Add an optional actions field (omitted when empty) to the structured element entry on all three platforms: - Linux: emit AtspiNode.actions in GetWindowStateTool, filtering blank names. - macOS: emit AXNode.actions in build_elements_array_with_token. The macOS implementation and tests were salvaged from PR trycua#2622. - Windows: emit UiaNode.actions in GetWindowStateTool. Each platform's tests now exercise the real production entry-building path rather than re-creating JSON by hand, and all three tool descriptions list the new `actions` field. Validated with: - cargo fmt -- --check - cargo check -p platform-linux - cargo check -p platform-macos - cargo check -p platform-windows - cargo test -p platform-linux get_window_state_actions_tests macOS and Windows tests are written but cannot be run locally on Linux; they will be exercised by CI. Co-authored-by: Haoqing Wang <hqhq1025@users.noreply.github.com>

…oss platforms The Linux, macOS, and Windows accessibility walkers already collect action names, but none of the platform get_window_state tools surfaced them in the structured elements array. This caused callers to see missing/empty actions even though the underlying element was actionable. Add an optional actions field (omitted when empty) to the structured element entry on all three platforms: - Linux: emit AtspiNode.actions in GetWindowStateTool, filtering blank names. - macOS: emit AXNode.actions in build_elements_array_with_token. The macOS implementation and tests were salvaged from PR trycua#2622. - Windows: emit UiaNode.actions in GetWindowStateTool. Fix a macOS regression where observation-only snapshots emitted unregistered element_token values: build_elements_array_with_token now takes Option<u32> and only emits element_token for Some(snapshot_id). Each platform's tests exercise the real production entry-building path, and all three tool descriptions list the new `actions` field. Validated with: - cargo fmt -- --check - cargo check -p platform-{linux,macos,windows} - cargo test -p platform-linux get_window_state_actions_tests - cargo test -p platform-macos (native macOS, 344 passed) - cargo test -p platform-windows (Windows 11 VM, 202 passed; launch_uwp flake) - Real Windows app check: Notepad get_window_state returns actions. Co-authored-by: Haoqing Wang <hqhq1025@users.noreply.github.com>

Expose AT-SPI/AX/UIA action names in the structured `elements` array of `get_window_state` on Linux, macOS, and Windows. The action list is omitted when empty, and Linux filters blank/whitespace-only names. macOS observation-only snapshots (`_observation_only`) continue to emit action names but do not emit `element_token`, since no snapshot is registered in that path. Refs trycua#3376. Salvaged from trycua#2622 (macOS). Co-authored-by: Haoqing Wang <hqhq1025@users.noreply.github.com>

Expose AT-SPI/AX/UIA action names in the structured `elements` array of `get_window_state` on Linux, macOS, and Windows. The action list is omitted when empty, and Linux filters blank/whitespace-only names. macOS observation-only snapshots (`_observation_only`) continue to emit action names but do not emit `element_token`, since no snapshot is registered in that path. Refs #3376. Salvaged from #2622 (macOS). Co-authored-by: Haoqing Wang <hqhq1025@users.noreply.github.com> (cherry picked from commit fc2160b) Salvaged from #3377.

…3617) * feat(cua-driver): expose action names in get_window_state elements Expose AT-SPI/AX/UIA action names in the structured `elements` array of `get_window_state` on Linux, macOS, and Windows. The action list is omitted when empty, and Linux filters blank/whitespace-only names. macOS observation-only snapshots (`_observation_only`) continue to emit action names but do not emit `element_token`, since no snapshot is registered in that path. Refs #3376. Salvaged from #2622 (macOS). Co-authored-by: Haoqing Wang <hqhq1025@users.noreply.github.com> (cherry picked from commit fc2160b) Salvaged from #3377. * docs(cua-driver): regenerate action reference on macOS --------- Co-authored-by: Wangxiaoxiaoa <102247755+Wangxiaoxiaoa@users.noreply.github.com>

hqhq1025 requested a review from f-trycua as a code owner July 28, 2026 07:23

hqhq1025 mentioned this pull request Jul 28, 2026

fix(cua-driver): carry full 32-bit snapshot generation in element tokens #2608

Merged

This was referenced Jul 29, 2026

fix(cua-driver): refuse stale element targets in press_key and type_text_chars #2672

Open

feat(cua-driver): add stable AX node identity tokens #2210

Closed

fix(cua-driver): invalidate element tokens when the same window is re-snapshotted #2673

Closed

injaneity mentioned this pull request Aug 21, 2026

fix(cua-driver): expose AT-SPI actions in get_window_state elements #3307

Closed

6 tasks

This was referenced Aug 25, 2026

[Feature]: get_window_state structured elements should expose action names across platforms #3376

Open

feat(cua-driver): expose action names in get_window_state elements #3377

Closed

injaneity mentioned this pull request Sep 7, 2026

feat(cua-driver): expose action names in get_window_state elements #3617

Merged

Navigation Menu

Uh oh!

Uh oh!

feat(cua-driver): expose element AX actions in structured elements - #2622

feat(cua-driver): expose element AX actions in structured elements#2622
hqhq1025 wants to merge 1 commit into
trycua:mainfrom
hqhq1025:codex/expose-element-actions

hqhq1025 commented Jul 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 28, 2026

Uh oh!

hqhq1025 commented Jul 28, 2026

Uh oh!

hqhq1025 commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

hqhq1025 commented Jul 28, 2026

Summary

This is the same gap that was already closed for value

Change

Platform scope

Test

Note

Uh oh!

chatgpt-codex-connector Bot commented Jul 28, 2026

Uh oh!

hqhq1025 commented Jul 28, 2026

Uh oh!

hqhq1025 commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

This is the same gap that was already closed for `value`